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ABSTRACT 


A  basic  approach  to  the  problem  of  evaluating  or  predicting  a  crew's 
performance  for  the  VT  community  is  presented.   The  method  uses  an  ap- 
plication of  multiple  regression  analysis  techniques  to  a  model  which 
has  training  parameters  as  its  variables. 

The  results  would  allow  the  squadron  or  wing  commanding  officer  to 
predict  a  crew's  performance  before  the  actual  flight  and  to  determine 
how  to  allocate  training  time  for  the  squadron. 
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I.   INTRODUCTION 

A.   BACKGROUND 

The  patrol  squadron  ASW  (VP)  community  has  been  trying  to  determine 
a  method  of  evaluating  crew  performance  effectively  for  many  years.   They 
not  only  would  like  to  evaluate  a  performance  by  a  crew  but  also  to  pre- 
dict a  level  of  performance  by  a  crew  before  the  flight.   The  advantages 
of  having  such  a  system  are  obvious.   A  continuing  performance  chart  of 
each  crew,  in  each  squadron  would  be  available.   With  such  a  device 
WINGSPAC  might  be  able  to  do  away  with  yearly  Operational  Readiness  In- 
spections (ORI) .   These  evaluations  have  been  used  to  check  the  ability 
of  the  squadron  to  meet  any  possible  contingency.   It  was  given  through 
a  multitude  of  flight,  written  examinations,  and  ground  evolutions. 
Basically  it  tested  the  squadrons  training  program  and  determined  if  a 
squadron  was  ready  for  deployment.   By  having  these  up  to  date  performance 
charts  of  all  crews  the  Wing  could  tell  at  any  time,  not  just  at  a 
scheduled  one  (as  with  the  ORI),  if  that  squadron  was  up  to  par. 

On  this  same  line  the  Wing  commander  could  see  from  these  charts  if 
a  certain  squadron's  overall  performance  trend  was  decreasing.   If  this 
was  accompanied  by  a  heavy  period  of  operational  tasking  it  could  mean 
the  squadron  should  be  stood  down  from  operational  flying  so  it  could 
devote  \nore  time  to  training.   The  problem  of  over  tasking  a  squadron 
with  operational  commitments  is  a  serious  problem  today  facing  the  Wing 
commander. 

In  addition  the  squadron  or  Wing  commander  would  be  able  to  determine 
before  the  flight  if  a  particular  crew  is  qualified  to  go  on  that  mission 
and  do  a  job  of  a  required  degree  of  skill  and  competence. 


B.   NATURE  OF  THE  PROBLEM 

At  the  present  time  squadrons  in  WINGSPAC  are  using  a  postflight 
evaluation  method  in  an  attempt  to  evaluate  a  crew's  performance.  The 
crew  is  evaluated  by  a  team  of  debriefers  from  the  various  ASCAC'S  upon 
completion  of  the  flight.  The  form  is  quite  lengthy  and  requires  con- 
siderable time  to  complete  during  the  debrief  and  after.  The  range  of 
grades  goes  from  0  to  100  for  each  area.  The  number  of  required  areas 
to  be  graded  being  dependent  on  the  type  of  flight  flown. 

The  major  problem  here  is  the  standardization  of  the  evaluators  in 
how  they  grade.   The  variance  in  their  attempts  for  objectivity  in 
quantifying  quality  has  been  extremely  high.   Another  area  for  concern 
is  the  length  of  the  form.   This  combined  with  the  large  number  of 
flights  graded  and  relatively  small  number  of  evaluators  available  causes 
a  rushed  evaluation  with  not  enough  emphasis  on  constructive  criticism. 
Last  but  not  least  is  the  fact  that  the  evaluators  are  not  completely 
devoid  of  personal  connections  to  the  people  they  are  evaluating. 

Because  of  the  above  problems  the  average  grade  of  a  sample  of 
flights  of  crews  from  several  squadrons  at  Moffett  Field  is  the  un- 
believably high  96-97%  and  is  attacked  by  WINGSPAC  and  the  Tactical 
Support  Center  (TSC)  at  Moffett  Field  now. 

The  problem  that  is  attacked  here  is  that  of  predicting  a  crew's 
performance.   This  problem  is  difficult  because  of  the  large  number  of 
variables  affecting  performance  and  the  lac   of,  in  some  cases,  of  a 
suitable  measure  of  that  variable. 

Because  of  this  it  was  decided  to  start  by  looking  strictly  at  the 
major  variables  associated  with  training.   The  amount  and  type  of  train- 
ing might  vary  between  squadrons  but  it  is  recognized  as  being  a  major 
necessity  for  all  VP  crews. 
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A  basic  approach  using  multiple  regression  analysis  was  then  under- 
taken with  the  selected  training  variable  as  independent  parameters  and 
the  postflight  evaluation  score  as  the  dependent  variable. 

The  inability  of  the  squadrons  to  collect  data  in  the  required 
format  did  not  allow  us  to  use  real  world  data  to  test  proposed  models. 
Data  was  generated  by  the  author,  however,  to  demonstrate  the  techniques 
involved.   The  data  used  is  a  reasonable  attempt  to  simulate  the  actual 
data,  based  on  prior  experience. 


II.      METHODOLOGY 

A.      THE  MODEL 

The  independent  variables  chosen  to  be  looked  at  are: 

X~   :    Total  flight  time  -  the  time  in  hours 

flown  as  a  crew  since  the  crew  was  formed, 

X~   :    Weapons  Systems  Trainer  (WST)  time  -  time 

in  hours  in  trainer  as  a  crew. 

X,   :    Sub  time  -  time  in  hours  of  actual  on  station 
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time  working  a  submarine  as  a  crew. 
The  dependent  variable  is: 

X1   :    Postflight  evaluation  score  -  overall 

grade  of  evaluators  given  as  a  percent. 
It  is  expected  that  the  value  of  X.  would  increase  with  an  increase 
in  the  independent  variable.   The  rate  of  increase  should  decrease  as 
the  independent  variables  get  larger.   One  method  of  mathematically 
describing  such  a  relationship  is  given  by: 

X   =   1  -  3e"(CV2X2  +Q'3X3+Q'4X4)  (1) 


where 


0  <;  X.  £  1 


0  £  p  ?;  1 


X2 '  X3 '  X4  ~ 


Let  Z  =  ff„X„  +  Q'  X~  +  o/.X,.   In  equation  (1)  as  the  value  of  Z 
11  5   5  q  4 

approaches  infinity  then  the  value  of  X,  approaches  1  because  the  value 


-co 


of   e        =0.       Similarly   as    the   value   of      Z      approaches      0      then   the   value 
of   X1    approaches  |3    because   e        =   1. 


This  relationship  is  shown  graphically  in  Figure  (1) 


1- 


0" 


> 


Figure   (1) 


B.   APPLICATION 

Before  the  model  could  be  used  in  a  linear  regression  analysis  it 
had  to  be  transformed  from  its  non-linear  form  in  equation  (1)  into  a 
linear  form.   This  can  be  done  by  using  a  logarithmic  transformation  on 
equation  (1).   Regrouping  terms  equation  (1)  becomes 


1  -  X, 


-(or0X0  +  or,,  X-  +  a,X, ) 
e   2  2    3  3    4  4 


(2) 


Now  applying  the  transformation  by  taking  the  natural  logarithm  of 
both  sides  equation  (2)  becomes 


y  =   In  (1  -  X  )   =   ln3  -  q'2X2  -  o/3Xr3  -  a.X, 


(3) 


The  natural  base  for  the  logarithm  was  chosen  for  convenience  only. 

In  this  form  it  is  ready  to  be  used  in  conjuntion  with  any  available 
computer  multiple  regression  package.   The  author  has  used  the  BIMED02R 
stepwise  linear  regression  package  [Ref.  l]  on  the  IBM  360  computer. 
This  program  allows  variables  to  enter  one  at  a  time  and  permits  the 
user  to  see  in  what  order  and  with  what  effect  each  does. 

For  manual  calculations  let  b  be  a  vector  containing  the  estimators 
of  the  constant  term  lnf3 ,  a~,   <y„,    and  a ,  .   Let  y   -   lnj3 . 


—            — 

y 

A. 

Qf2 

a3 

A. 

a4 

-                    m 

where  the  symbol      indicates  that  a  quantity  is  an  estimator, 


Let  x  be  a  matrix  of  the  observations  on  all  independent  variables 
augmented  by  a  column  of  units  representing  the  constant  term. 


i  x12  x13  x14 
l  x22  x23  x24 


X  = 


I     X   r>     X   0     X  , 

n2   nJ   n4 


and  x'   ^s  X  transpose 


X'  - 


X12  X22  X32 
X13  X23  X33 
X14  X24   X34 


'n3 


X 


n4 


Let  y  be  a  vector  of  observations  of  the  dependent  variable, 


Y  = 


The  least  squares  estimates  are  given  in  matrix  notation  by 

A. 

b  -   (X'X)"1  X'Y 


(A) 


If  the  problem  was  reduced  to  only  one  independent  variable,  equation 
(3)  would  simply  be 


In  (1  -  Xx)   -   lnp   -  cv2X2 
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or  by  some  basic  substitutions 

y   =  A  +   BX 

where   A  =   lnft   and   B  =   -  Q?„ 

Solving  for  the  estimates  of  A  and   B  by  linear  regression  the  re- 
sults are: 

A  =   y   -   BX 

n 

£  (X  -  X)  (y   -  y) 

B   =  i=i 

n 

£  (X.  -  2T) 
i=l 

where  the  _  denotes  the  mean  value  of  all  observations  of  that  variable, 

A  point  estimate  of  y   given  a  particular  value  of  X  can  be 

found  from  the  equation 

A  A.   . 

y  =  A  +  BX*" 

where  y   is  the  point  estimate  and   X    is  a  particular  value  of  X. 
To  use  the  t-test  we  must  solve  for  the  standard  deviation  of  B. 
This  is  given  by 


S  (y±  -   y)2 


n  -  2  £  (X.  -  X)2 


The  confidence  interval  for   B   is 


Bit      ,  aZ 

c,2,n-l   B 
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C.   DATA  COLLECTION 

Data  has  to  be  collected  by  the  fleet  to  test  the  model.   It  could 
be  done  in  the  following  manner.   For  any  flight  that  is  to  be  graded 
the  tactical  coordinator  of  the  crew  can  give  the  values  of  the  variables 
X0 ,  X„ ,  X.  either  during  the  brief  or  the  debrief.   After  the  evaluators 

Z     J     H 

have  finished  grading  the  flight  they  will  record  the  overall  score  as 

variable  X1  and  turn  into  collection  point  along  with  squadron,  crew, 

event  number,  and  date.   These  values  of  X. ,  X0 ,  X0 ,  and  X.  form  one 

1    I        5  A- 

set  of  observations.   When  a  reasonable  number  have  been  gathered  they 
can  be  used  with  the  regression  package  and  run  on  the  computer  or  using 
formulas  (4)  and  a  calculator  the  model  can  be  tested  and  the  estimators 
found.   Once  the  estimator  of  the  constant  ln(3  and  the  estimators  of  the 
coefficients  &„,  <y~,    and  or,  have  been  determined  these  values  can  be 
used  to  predict  future  values  of  the  dependent  variable   X. . 
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III.   DATA   ANALYSIS 


Summary  of  computer  results  using  data  in  Table  1 

coefficient 
X.  -.00105 


x3 

X4 
constant 

multiple  R 

R  square 


-.00698 
.00172 

-.97327 
.8859 
.7848 


standard  error 
.00076 

.00272 

.00994 


A  t-test  is  used  to  test  the  hypothesis  that  the  value  of  o.'.   is 
equal  to  0.   The  hypothesis  is  accepted  if 

i  =  2,3,4 


0! . 
"  t        .   £   -J-   <   t        . 
C/^"1     cr^       c/2'n"1 


72 


a 


and  rejected  if    i     lies  outside  this  interval.   In  the  avovc  equation 

c  is  equal  to  1-  confidence  level  desired  and  n-1  is  called  the  degrees 
of  freedom,  where  is  the  number  of  observations. 

A  t-table  can  be  found  in  [Refs.  2,3,4,  and  5]  and  many  other 
textbooks.   Part  of  a  typical  table  is  shown  below: 


df.    =   n-1       >^ 
30 

.1 

.05 

.025 

1.310 

1.697 

2.042 

40 

1.303 

1.684 

2.021 

50 

1.298 

1.676 

2.009 

One  would  enter  with  the  degrees  of  freedom  (df)  equal  to  n-1  down 
the  left  column  and  the  value  of  c/2  across  the  top. 
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For  a  907u  confidence  level  for  the  given  data  we  would  enter  with 
df  =  39  and  c/2  =  .05.   The  value  obtained  was  1.685 

i  cot  .00105      .  ,Q, 

~    X'684  *     -     700076   *   U684 

-  1.684  <  -   1.381    <  1.684 

This  tells  the  user  that  the  coefficient  is  not  significant  at  the 
90%  confidence  level  or  that  it  is  equal  to  0  at  this  level.   It  can  be 
seen  from  the  t-tables  that  it  becomes  significant  at  the  80%  level.   The 
t  value  being  1.303. 

A. 

Similarly  & „  can  be  tested.   Using  a  90?o  confidence  level 

i  ro,  .00698      .  ,„. 

"  X-684  *   "  700272   *  l'6BU 

-1.684   <  -  2.564    <  1.684 

/s 

Here  a-    is  significant  and  different  from  0. 

Just  by  looking  at  the  computer  results  for  a,    it  can  be  seen  that 
the  standard  error  is  considerably  larger  than  the  coefficient  itself. 

A 

This  indicates  that  a,    is  not  significant  at  any  practical  level  of  con- 
fidence. 

Another  important  figure  in  the  results  is  the  R  square.   This  in- 
dicates that  78%  of  the  variance  in  the  data  is  accounted  for  by  the 
model  and  its  significant  variables. 

Since  a  logarithmic  transformation  was  used  on  the  model  any  term 
in  the  final  results  so  affected  has  to  be  transformed  back  to  its 
usable  form  in  the  model.   Here  only  the  constant  term  is  affected. 
Taking  the  antilog  of  -0.97327  with  a  natural  base  the  resulting  value 
is  .3779. 
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The  equation  for  predicting  future  scores  would  be 
Xl  =  1  -  .3779o-<-00105X2  +  -00698V 

A.  A. 

With  the  data  used  the  values  of  a~      and  a„    indicate  the  relative 
eighing  factors  of  the  training  parameters   X„  and  X^. .   Since  <y »   =    .00105 


and  a~    =  .00698,  it  would  indicate  that  the  training  denoted  by  X„  would 
have  approximately  6.5  times  the  effect  on  performance  as  the  training 
denoted  by  X~ .   Again  this  is  strictly  for  the  data  used  in  this  analysis 
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IV.   CONCLUSIONS 

The  approach  to  the  problem  is  only  a  basic  feasible  starting  point, 
The  model  presented  may  be  one  of  many  possible  ones.   The  fleet  has  to 
take  the  model  presented  and  test  it  with  actual  data  collected  on  a 
higher  authority  basis,  using  the  methodology  presented  here.   The  num- 
erical results  found  here  do  not  apply  to  the  fleet  because  they  were 
arrived  at  using  generated  data.   The  implications  from  the  tests  show 
that  the  model  could  work  and  should  be  tested.   Once  this  is  done  the 
estimators  that  are  found  can  be  used  to  predict  the  performance  of  any 
crew  on  a  pending  flight. 

An  important  secondary  use  of  these  coefficient  estimators  is  as 
a  weighting  factor  to  determine  allocation  of  training  time  within  a 
squadron.   The  coefficients  form  a  relative  merit  scale  of  each  parameter 
used  in  the  regression. 

The  user  can  add  as  many  independent  variables  as  he  chooses  with- 
out complicating  the  methodology.   Since  the  BIMED02R  is  a  stepwise 
regression  model,  the  user  is  able  to  determine  the  effect  of  these 
variables  individually  if  he  can  collect  the  data  in  a  usable  form. 
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APPENDIX  A 

DATA  USED  FOR  SAMPLE  ANALYSIS 

OBS  NOc  X2  X3  X^ 

1  231  20  12 

2  20  8  0 

3  30  8  4 

4  38  11  4 

5  54  11  4 

6  64  14  4 

7  830  152  64 

8  842  152  64 

9  850  152  64 

10  878  155  68 

11  902  158  68 

12  453  45  38 

13  461  48  J>3 

14  490  51  2 

15  498  54  42 

16  513  57  42 

17  240  23  16 

18  252  26  16 

19  261  26  20 

20  270  29  20 

21  136  20  4 

22  492  86  48 

23  316  72  44 
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OBS  NO, 


X, 


x, 


24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 


224 
84 
524 
185 
381 
204 
523 
620 
450 
362 
180 
532 
1226 
239 
781 
796 
93 


44 
20 
120 
45 
57 
12 
60 
120 
72 
28 
24 
100 
147 
21 
68 
106 
18 


24 

1 
40 

8 
24 

4 
36 
40 
32 
20 
12 
40 
83 
16 
54 

'.8 
7 
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parameters  as  its  variables. 

The  results  would  allow  the  squadron  or  wing  commanding  officer  to 
predict  a  crew's  performance  before  the  actual  flight  and  to  determine 
how  to  allocate  training  time  for  the  squadron. 
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